A novel anti-loosening bolt looseness diagnosis of bolt connections using a vision-based technique

Bolt looseness detection is a common problem in engineering. Most vision-based detection techniques focus on diagnosing ordinary bolt looseness, i.e., the methods used for diagnosis are based only on the sidelines of nuts. These methods cannot be used for anti-loosening bolt looseness diagnosis because of the simultaneous rotation of screws and nuts. Therefore, a novel anti-loosening bolt looseness diagnosis method based on a vision-based technique is proposed in this paper. First, a regular hexagonal cap was installed on the screw, which can be used as a reference for the nut. Then, to automatically distinguish the hexagonal borders of the screw cap and nut, a new hexagonal border reconstruction algorithm is proposed. Furthermore, the relative rotation angles of the screw cap and nut hexagons can be determined using the sidelines of the reconstructed hexagonal borders of the screw cap and nut. Finally, a novel anti-loosening bolt looseness diagnosis method is established by using the relative rotation angle of the regular hexagonal borders of the screw cap and nut under initial status and loose status. A prototype flange node of the transmission tower was used for experimental verification. The results show that the proposed method can effectively detect the loosening angle of anti-loosening bolts.

technology, the edge lines of nuts can be identified.The angles between these edge lines and the horizontal coordinate axes are related to the loosening angle of the mainstream regular hexagonal bolts 10,11 , which can be used to quantify the loosening angle.The vision-based bolt loosening detection method is stable and has high detection accuracy 12 .With the rapid development of neural networks, deep learning and image processing technology have been combined to improve the efficiency of bolt location identification 13 , detect the integrity of bolts, and classify loose and nonloose bolts [14][15][16][17] .Breakthroughs in deep learning technology have greatly improved the accuracy and efficiency of bolt looseness detection methods based on vision technology, greatly increasing their application value.
However, there are still certain shortcomings in most vision-based detection methods, i.e., anti-loosening bolt looseness cannot be detected quantitatively.The reason is that the current quantitative detection methods for bolt looseness using a vision-based technique are almost exclusively based on the sideline angles of nuts.As shown in Fig. 1, under loose state 1, only the nut of the bolt rotates clockwise, and the rotation angle is 30 degrees.Therefore, using the existing quantitative detection method of bolt looseness based on vision technology, the rotation of the nut can be detected, and the conclusion that the bolt is loosened by 30 degrees can be reached.However, for anti-loosening bolts, the screw and nut may rotate together.For example, under loose state 2 in Fig. 1, the nut and screw rotate 30 degrees synchronously, and the bolt is actually not loose because there is no relative rotation between the screw and nut.Based on the existing method, the same conclusion as loose state 1 will be given, which is an incorrect conclusion.Hence, the relative rotation between the screw and nut will have a negative impact on the identification of the loosening angle.
To this end, a novel anti-loosening bolt looseness diagnosis method is proposed to quantitatively detect anti-loosening bolt looseness.First, a regular hexagonal cap was installed on the screw, which can be used as a reference for the nut.Then, to automatically distinguish the hexagonal borders of the screw cap and nut, a new hexagonal border reconstruction algorithm is proposed.Furthermore, the relative rotation angles of the screw cap and nut hexagons can be determined using the sidelines of the reconstructed hexagonal borders of the screw cap and nut.Finally, a novel anti-loosening bolt looseness diagnosis method is established by using the relative rotation angle of the regular hexagonal borders of the screw cap and nut under initial status and loose status.
The main contributions of this paper are summarized as follows: 1.A novel anti-loosening bolt looseness diagnosis method is proposed for anti-loosening bolt looseness detection.The proposed method can quantitatively detect bolt loosening with high detection accuracy and low cost.2. A new hexagonal border reconstruction algorithm is proposed for automatically distinguishing the hexagonal borders of the screw cap and nut.The centroid of the hexagonal cap is used as the reference point, and the distances from the endpoints on the edge lines of the hexagonal borders of the screw cap and nut to the reference point are used as the clustering constraints.Then, the intra-class variance algorithm is introduced to automatically distinguish the edge lines of the hexagonal borders of the screw cap and nut, and the hexagonal borders of the screw cap and nut can be reconstructed.3. The proposed method was verified to be effective, and the potential influencing factors were also analyzed.
The results show that the proposed method in this paper can more comprehensively detect anti-loosening bolt looseness with high accuracy and good stability.
The remainder of this paper is arranged as follows.Related works on bolt loosening detection are presented in "Related Work".The "Proposed Method" section mainly introduces the theory of the proposed novel antiloosening bolt looseness diagnosis method.The "Experiments" section mainly verifies the feasibility of the proposed method, including method validation and analysis of the influencing factors.The "Conclusion" section contains the conclusion.

Bolt loosening diagnosis based on hardware sensors
Due to the inefficiency of traditional manual maintenance of bolt loosening, the detection accuracy is inadequate.In recent years, bolt looseness detection methods based on hardware sensors have attracted widespread attention from researchers.An et al. 18 proposed a bolt loose detection method based on integrated impedance and guided waves, which utilizes piezoelectric transducers to obtain impedance and guided wave signals for bolt loose detection.Yin et al. 6 proposed a bolt looseness detection method based on the piezoelectric active sensing method, which determines the actual contact area of bolts through ultrasonic energy transmission, thereby determining the looseness of bolts.Zhao et al. 19 proposed a detection method for timber structural bolts based on wavelet analysis that combines the pasting of lead zirconate titanate with the time-recursive method.Zhang et al. 20 proposed a bolt looseness detection method based on audio classification using a support vector machine, which mainly records and extracts the hammer sound produced by the varying looseness of bolt connections.Wang et al. 21proposed a vibroacoustic method for detecting bolt looseness.The above methods can achieve unmanned detection and sufficient detection accuracy.However, some of these sensors are relatively expensive.

Bolt loosening diagnosis based on visual technology
Currently, with the improvement of image processing technology, vision-based techniques have been introduced for bolt looseness detection because of their lower cost, greater intelligence and greater accuracy.Ramana et al. 22 proposed a fully automatic bolt looseness detection method based on the Viola-Jones algorithm and a support vector machine.The Viola-Jones algorithm was used to determine the bolt location, and a support vector machine was used to classify the loose bolts.Park et al. 10 and Nguyen et al. 11 proposed a bolt connection looseness detection technology-based Canny edge detection algorithm 8 and Hough transform 9 , which identifies bolt loosening by comparing the sideline angles before and after bolt loosening.Kong et al. 12 proposed a visual noncontact bolt looseness detection method by judging whether the bolt alignment process has rotated between two bolt images at different times.Cha et al. 23 proposed a loose bolt detection method using the Hough transform and support vector machine, which can automatically calculate the damage sensitive features of bolts to train the support vector machine and construct a robust classifier to distinguish tight bolts and loose bolts.Luo et al. 24 proposed a new bolt image correction method based on a square gasket for flange connection node bolts.
Additionally, with the rapid development of deep learning, vision-based techniques have become more intelligent and efficient.The main methods include deep learning only and the combination of deep learning and image processing technology [13][14][15][16][17][18][19][20][21][22][23][24] .Huynh et al. 13 used convolutional neural networks (CNNs) 25 to identify bolts and segment bolt images, and the nut edge line angles from the initial and current states were used to diagnose bolt looseness based on image processing technology.Zhang et al. 15 used the Faster-RCNN 26,27 deep learning algorithm for autonomous bolt looseness detection based on different screw heights, and the experimental results showed that the average accuracy of looseness detection reached 0.9503.Yang et al. 16 comparatively tested the detection performance of three YOLO target detection algorithms 28 for bolt loosening identification, namely, YOLO v3, YOLO v4 and YOLO v4-Tiny.Yuan et al. 29 proposed an automatic bolt loosening identification method based on Mask-RCNN 30 .Gong et al. 31 proposed a quantitative bolt loosening identification method by calculating the exposed length of the screw using vision-based deep learning and geometric imaging theory, and the results showed that the average measurement error was limited to within 0.61 mm.However, the relationship between the loosening angle and the exposed length of the screw is still unclear.Pan et al. 32,33 proposed a bolt looseness detection method based on 3D vision.The bolts in the 3D model are located and extracted based on the CNN convolutional neural network and point cloud processing algorithm, and the height of the screw is used to determine whether the screw is loose.
The above studies mainly focused on the quantitative nut rotation and the exposed height of the screw.However, less research has focused on identifying loosening of anti-loosening bolts.Therefore, in this paper, a new anti-loosening bolt loosening detection method is proposed.

Bolts with regular hexagonal caps
The vertical and top views of the proposed anti-loosening bolt construction in this paper are shown in Figs. 2 and  3, respectively.An additional regular hexagonal cap was installed at the end of the screw.The distance between the screw cap and nut hexagons was 7.5 mm, as shown in Fig. 2. Except for the regular hexagonal cap, the other bolts are the same as the normal anti-loosening bolt, including the screw, nut, circular washer, and connected components.The regular hexagonal cap is white, and the nut is black, which can maximize the gray gradient difference between the screw cap and nut edge 34 and improve the stability of edge detection.

The proposed hexagonal border reconstruction algorithm
When photos of the anti-loosening bolt are obtained using a camera at the top, the hexagonal borders of the screw cap and nut can be determined using the Canny edge detection method, as shown in Fig. 4.However, the relative rotation angle between the screw cap and nut is the desired quantity.Therefore, a new hexagonal border reconstruction algorithm must be proposed to automatically distinguish the hexagonal borders of the screw cap and nut.
The proposed hexagonal border reconstruction algorithm is shown in Fig. 5. First, the initial image is transformed into grayscale, denoised by a Wiener adaptive filter 35 , and binarized by the Otsu method 36,37 .Second, connected domain processing and morphological operations are used, and the morphological operations usually include erosion, expansion, deletion of interfering binary image areas, calculation of binary image centroid coordinates, etc.Then, the interference areas are deleted, and the centroid coordinates of the regular hexagonal Vol:.( 1234567890 cap can be obtained.Third, the edge lines of the nut and regular hexagonal cap can be segmented.The Canny algorithm 8 and Hough algorithm 9 are used to detect edge lines on the initial bolt image, and the endpoint coordinates of the straight edge lines can be extracted.Then, the distances between all endpoint coordinates and the identified centroid point of the regular hexagonal cap can be calculated, and the intra-class variance method [38][39][40] is used to adaptively calculate the optimal segmentation distance, which can be used to segment the edge lines of the nut and the regular hexagonal cap.Finally, the hexagonal borders of the nut and regular hexagonal cap can be reconstructed using individual edge lines.Since the Otsu threshold segmentation method can maximize the difference between the front and rear backgrounds and is not affected by image brightness, the Canny algorithm is not sensitive to image brightness.Therefore, the program in this section is not affected by image brightness and has high stability.www.nature.com/scientificreports/

Relative rotation angle of the screw cap and nut hexagons
After reconstructing the hexagonal border of the screw cap or nut, the rotation angle of the screw cap or nut hexagon can be determined.As shown in Fig. 6, a regular hexagonal border has six edges parallel to each other, so there are only three angles on the x-o-y coordinate system, namely, θ 1 , θ 2 , and θ 3 .θ 1 , θ 2 , and θ 3 have the following angular relationships: Therefore, when the hexagonal border starts to rotate, the angles θ 1 , θ 2 and θ 3 will undergo changes.Thus, based on the relationships among these three angles, the rotation angle of the hexagonal border can be calculated using the angles θ 1 , θ 2 and θ 3 as follows: where θ j is the angle between the j th edge line and the x-axis, n is the number of identified sidelines, and rem(.) is the operator used to determine the remainder after division.
Based on Eqs. ( 1) and ( 2), the rotation angle of the hexagonal border of the screw cap can be calculated and defined as θ inner .Similarly, the rotation angle of the hexagonal border of the nut can be calculated and defined as θ out .Then, the relative rotation angle of the screw cap and nut hexagons can be defined as

The proposed anti-loosening bolt looseness diagnosis
When the bolt is not loose, the relative rotation angle of the hexagonal borders of the screw cap and nut can be determined and defined as θu .Meanwhile, when the bolt is loose, the relative rotation angle of the hexagonal (1)  borders of the screw cap and nut can be determined and defined as θd .Then, the bolt looseness angle of the anti-loosening bolt can be expressed as: In other words, the proposed method in this paper involves the installation of a regular hexagonal screw cap, the reconstruction of hexagonal borders, the identification of the relative rotation angle of the screw cap and nut, and the comparison of the relative rotation angles under the initial status and loose status.The process of the proposed anti-loosening bolt looseness diagnosis method is shown in Fig. 7.

Experimental Experimental overview
To verify the feasibility of the proposed method, a laboratory experiment is implemented.A prototype flange node of the transmission tower was used, as shown in Fig. 8.The hexagonal cap and the screw were connected by an adhesive.One of the bolts was selected for study, and images were captured using an iPhone 12 in the laboratory.
In the experiment, one initial bolt status is considered.Four bolt looseness cases were considered, i.e., the relative rotation angles of the screw cap and nut hexagons were 0°, 10°, 20°, and 30°.For the 0° bolt looseness case, the screw and nut rotate synchronously, and the location of the screw cap and nut is not the same as that (4) θ * = θd − θu .www.nature.com/scientificreports/for the 0° initial status, as shown in Fig. 7.For the 10°, 20°, and 30° bolt looseness cases, only the nut rotation and looseness angles are simulated by using the change in the relative rotation angle between the screw cap and nut, as shown in Fig. 7. To set the angles accurately, seven adjacent 10-degree scale lines were drawn on the gasket, as shown in Fig. 7.
The influences of lighting intensity and shooting distance were also studied.Seven lighting intensities, four shooting heights and twelve shooting perspective angles are considered and are discussed in the "Analysis of influencing factors" section.

The validation experiment of the proposed method
Images obtained under a lighting intensity of 200 lx and a shooting height of 50 cm were used to verify the proposed method.Under the initial status, one image was captured to calculate the quantity θu .Under each looseness case, twenty images were captured to study the stability of the identification results.Twenty images under each looseness case were processed, and twenty relative rotation angles θd between the screw cap and nut were obtained.Examples of edge line segmentation results under different bolt states are shown in Fig. 9.
Then, based on Eq. ( 4), twenty looseness angles θ* can be calculated for each looseness case, as shown in Fig. 10.Ave represents the average value of 20 test results per group.Std represents the standard deviation of 20 test results per group.The results in Fig. 10 show that the mean value deviations are small, with a maximum of only 0.62° under the 30° looseness case.The standard deviations do not exceed 0.54° under the 30° looseness case.The proposed method has good effectiveness and feasibility and high accuracy.

Comparison of the proposed method and existing methods
One of the images under the 0° initial state and 0° looseness state are used to compare the proposed method and existing bolt looseness quantitative identification methods 13 .To make the calculation process more intuitive, the calculation results obtained using the two methods are shown in Fig. 11.Based on the existing method, the results show that the bolt is already loose and that the rotation angle is 10.5°.In contrast, based on the proposed method, the results show that there is no loose bolt.The method proposed in this article can effectively avoid the impact of the synchronous rotation of the nut and screw on bolt looseness detection.

Analysis of the influencing factors
To study the influence of camera height, light intensity and shooting perspective angle, bolt looseness detection tests under different test conditions were implemented.Seven lighting intensities, four shooting heights and twelve shooting perspective angles are considered.www.nature.com/scientificreports/Based on the proposed method, the looseness angles under different shooting heights can be calculated, as shown in Fig. 12.The results show that the impact of shooting height is very limited.Generally, for all the shooting heights, the maximum mean deviation is 0.62° under the 30° looseness case, and the maximum standard deviation is 0.91° under the 10° looseness case, which are still within an acceptable range.However, it is not recommended to shoot bolts at a high height.The reason is that the lower the camera height is, the more details the camera captures, the better the edge identification, and the greater the stability of the bolt looseness detection.Based on the proposed method, the looseness angles under different lighting intensities can be calculated, as shown in Fig. 13. Figure 13 shows that there is not much difference between the mean value deviations under different light intensities, and the impact of light intensity is very limited.The mean deviation is limited to between -0.75° and 0.58°, which is within an acceptable range.However, the standard deviations under strong or dark  www.nature.com/scientificreports/light intensities are generally greater than those under normal light intensities, which means that there is greater fluctuation in the identified results under strong or dark light intensities.Under a strong lighting environment, the luminescence on the bolt interferes with imaging, reducing the stability of the proposed method.In a dark lighting environment, the number of noise points in the image will increase.Hence, it is not recommended to shoot images under extreme lighting environments, such as strong or dark lighting environments.In a strong lighting environment, the luminescence on the bolt interferes with imaging.In a dark lighting environment, the number of noise points in the image will increase.

Effect test and result analysis of different shooting perspective angles
The considered shooting perspective angles include horizontal perspective, vertical perspective, and horizontalvertical perspective angles.The horizontal and vertical perspective angles are set to 10°, 20°, 30° and 45°.The horizontal-vertical perspective angles are set to 10°-10°, 10°-30°, 30°-10° and 45°-45°.The looseness angles are set to 0°, 10° and 30°.The camera shooting height was set to 50 cm within an indoor environment.A series of 20 images were captured under each perspective angle and each looseness state.Figure 14 displays some bolt images with various perspective angles under a 10° looseness state.
Then, the collected images first undergo distortion correction, and the distortion correction method based on the homography matrix is used 41 .Finally, based on the corrected images and the proposed method in this paper, the looseness angles can be calculated, as shown in Figs. 15, 16 and 17.The average and standard deviation values of the results are shown in Fig. 18.The results show the following: (1) The perspective angle has a significant impact on the bolt looseness angle.The identification results using bolt images without perspective angle correction fluctuate more significantly.The maximum fluctuation is from the 0° looseness state and 45° horizontal perspective angle, where the maximum identification error reaches 6.5°.Furthermore, based on the standard deviation results, when the perspective angle is greater than 20°, the fluctuations in the identification results increase when using uncorrected bolt images.(2) The distortion correction method based on the homography matrix is effective.Using the corrected images, the fluctuation of the results is small.The maximum fluctuation is from the 0° looseness state and 45°-45° bidirectional perspective angle, where the maximum identification average value error is 1.1°.(3) In summary, the perspective angle has a significant impact on loosening angle identification.However, the distortion correction method based on the homography matrix is effective, and the impact of the perspective angle on the identification of loosening of the anti-loosening bolt can be effectively reduced.

Conclusion
This paper proposes a novel anti-loosening bolt looseness diagnosis method using a vision-based technique.
A regular hexagonal cap was installed on the screw.Then, a new hexagonal border reconstruction algorithm is proposed to automatically distinguish the hexagonal borders of the screw cap and nut.Finally, a novel antiloosening bolt looseness diagnosis method is established by using the relative rotation angle of the regular www.nature.com/scientificreports/hexagonal borders of the screw cap and nut under initial status and loose status.A prototype flange node is used to verify the proposed method.Four bolt looseness cases, four shooting heights, seven lighting intensities and twelve shooting perspective angles were considered in the experiment.The results show that the proposed method can effectively detect the looseness angle of anti-loosening bolts.Meanwhile, it should be noted that only the standard hexagonal bolts are concerned in this paper and the applicability of the method to other bolt feature needs further discussion.

Figure 2 .
Figure 2. Vertical view of the constructed bolt.

Figure 3 .
Figure 3. Top view of the constructed bolt.

Figure 4 .
Figure 4. Bolt picture and the hexagonal borders.

Figure 5 .
Figure 5. Process of the proposed hexagonal border reconstruction algorithm.

Figure 6 .
Figure 6.Three angles of a regular hexagonal border.

Figure 7 .
Figure 7.The process of the proposed anti-loosening bolt looseness diagnosis method.

Figure 10 .
Figure 10.Test results for the 20 bolt looseness cases.

Figure 11 .
Figure 11.Comparison of the proposed method and existing methods.
Seven light intensities are considered, i.e., 22,686 lx, 14,142 lx, 5172 lx, 350 lx, 200 lx, 14 lx and 7 lx.The lighting intensities correspond to outdoor strong sunshine weather, outdoor sunshine weather, outdoor bright cloudy weather, outdoor cloudy weather, indoor weather, outdoor evening, and outdoor night, respectively.The shooting height is set as 50 cm.The bolt images with different light intensities under 10° looseness state are shown in Fig. 13.Under each light intensity, 20 bolt images have been obtained.In order to better showcase its fluctuations under different light intensities, the first image under each light intensity is used as the initial state.Therefore, the looseness angle is actually set to 0°.The image acquisition and calculation of the looseness angles θ* are consistent with those in the previous section.

Figure 12 .
Figure 12. Results for different shooting heights.

Figure 14 .
Figure 14.Schematic diagram of bolt images from different perspective angles under a 10° looseness state.

Figure 15 .
Figure 15.Calculated bolt looseness angles from a horizontal perspective under different looseness states.

Figure 16 .
Figure 16.Calculated bolt looseness angle from the vertical perspective under the different looseness states.

Figure 17 .
Figure 17.Calculated bolt looseness angle from the horizontal-vertical perspective under the different looseness states.

Figure 18 .
Figure 18.Average and standard deviation values of the anti-loosening bolt loosening diagnostic results for different perspective angles.